Program Director Attitudes Toward Acceptance of Pediatric Dentistry Entrustable Professional Activities (EPAs).

Purpose: To acquire comments on pediatric dentistry entrustable professional activities (EPAs) from pediatric dentistry residency program directors (PDs). Methods: An electronic survey invited PDs to evaluate 16 previously developed EPAs on whether they were critical to patient safety, resident education, or both. PDs were asked to evaluate a fully developed EPA to assess structure and clarity and describe barriers to EPA. Descriptive statistics were completed. Results: Forty-one of 103 PDs completed the entire survey. Eighty-five percent (36 of 42) of PDs believed EPAs are critical to pediatric dentistry education, and 81 percent (34 of 42) believed EPAs are critical to patient safety. Eighty-one percent of PDs would likely use EPAs when available. Seventy-five percent (31 of 41) of PDs reported that they have had a resident who would have benefited from a longer duration of training. Conclusions: The majority of pediatric dentistry residency program director participants surveyed reported that entrustable professional activities are critical to patient safety and resident education. EPAs may be a valuable option for assessing residents' readiness for graduation.

The Next Era of Assessment: Building a Trustworthy Assessment System.

Assessment in medical education has evolved through a sequence of eras each centering on distinct views and values. These eras include measurement (e.g., knowledge exams, objective structured clinical examinations), then judgments (e.g., workplace-based assessments, entrustable professional activities), and most recently systems or programmatic assessment, where over time multiple types and sources of data are collected and combined by competency committees to ensure individual learners are ready to progress to the next stage in their training. Significantly less attention has been paid to the social context of assessment, which has led to an overall erosion of trust in assessment by a variety of stakeholders including learners and frontline assessors. To meaningfully move forward, the authors assert that the reestablishment of trust should be foundational to the next era of assessment. In our actions and interventions, it is imperative that medical education leaders address and build trust in assessment at a systems level. To that end, the authors first review tenets on the social contextualization of assessment and its linkage to trust and discuss consequences should the current state of low trust continue. The authors then posit that trusting and trustworthy relationships can exist at individual as well as organizational and systems levels. Finally, the authors propose a framework to build trust at multiple levels in a future assessment system; one that invites and supports professional and human growth and has the potential to position assessment as a fundamental component of renegotiating the social contract between medical education and the health of the public.

How do validity experts conceptualise argumentation? It's a rhetorical question.

INTRODUCTION: Health professions education (HPE) has adopted the conceptualization of validity as an argument. However, the theoretical and practical aspects of how validity arguments should be developed, used and evaluated in HPE have not been deeply explored. Articulating the argumentation theory undergirding validity and validation can help HPE better operationalise validity as an argument. To better understand this, the authors explored how HPE validity scholars conceptualise assessment validity arguments and argumentation, seeking to understand potential consequences of these views on validation practices. METHODS: The authors used critical case sampling to identify HPE assessment validity experts in three ways: (1) participation in a prominent validity research group, (2) appearing in a bibliometric study of HPE validity publications and (3) authorship of recent HPE validity literature. Qualitative semi-structured interviews were conducted with 16 experts in HPE assessment validity from four different countries. The authors used reflexive thematic analysis to develop themes relevant to their research question. RESULTS: The authors developed three themes grounded in participants' responses: (1) In theory, HPE validity is a social and situated argument. (2) In practice, the absence of audience and evaluation stymies the social nature of HPE validity. (3) Lack of validity argumentation creates and maintains power differentials within HPE. Participants articulated that current HPE validation practices are rooted in post-positivist epistemology when they should be situated (i.e. context-dependent), audience-centric and inclusive. DISCUSSION: When discussing validity argumentation in theory, participants' descriptions reflect an interpretivist lens for evaluation that is misaligned with real-world validity practices. This misalignment likely arises from HPE's adoption of "validity as an argument" as a slogan, without integrating theoretical and practical principles of argumentation theory.

A realist synthesis of prospective entrustment decision making by entrustment or clinical competency committees.

INTRODUCTION: The real-world mechanisms underlying prospective entrustment decision making (PEDM) by entrustment or clinical competency committees (E/CCCs) are poorly understood. To advance understanding in this area, the authors conducted a realist synthesis of the published literature to address the following research question: In E/CCC efforts to make defensible prospective entrustment decisions (PEDs), what works, for whom, under what circumstances and why? METHODS: Realist work seeks to understand the contexts (C), mechanisms (M) and outcomes (O) that explain how and why things work (or do not). In the authors' study, contexts included individual E/CCC members, E/CCC structures and processes, and training programmes. The outcome (i.e. desired outcome) was a PED. Mechanisms were a substantial focus of the analysis and informed the core findings. To define a final corpus of 52 included papers, the authors searched four databases, screened all results from those searches and performed a full-text review of a subset of screened papers. Data extraction focused on developing context-mechanism-outcome configurations from the papers, which were used to create a theory for how PEDM leads to PEDs. RESULTS: PEDM is often driven by default (non-deliberate) decision making rather than a deliberate process of deciding whether a trainee should be entrusted or not. When defaulting, some E/CCCs find red flags that sometimes lead to being more deliberate with decision making. E/CCCs that seek to be deliberate describe PEDM that can be effortful (when data are insufficient or incongruent) or effortless (when data are robust and tell a congruent story about a trainee). Both information about trainee trustworthiness and the sufficiency of data about trainee performance influence PEDM. Several moderators influence what is considered to be sufficient data, how trustworthiness data are viewed and how PEDM is carried out. These include perceived consequences and associated risks, E/CCC member trust propensity, E/CCC member personal knowledge of and experience with trainees and E/CCC structures and processes. DISCUSSION: PEDM is rarely deliberate but should be. Data about trainee trustworthiness are foundational to making PEDs. Bias, equity and fairness are nearly absent from the papers in this synthesis, and future efforts must seek to advance understanding and practice regarding the roles of bias, equity and fairness in PEDM.

How Does TIMELESS Training Impact Resident Motivation for Learning, Assessment, and Feedback? Evaluating a Competency-Based Time-Variable Training Pilot.

PURPOSE: As competency-based medical education has become the predominant graduate medical education training model, interest in time-variable training has grown. Despite multiple competency-based time-variable training (CBTVT) pilots ongoing in the United States, little is known about how this training approach impacts learners. The authors aim to explore how their CBTVT pilot program impacted resident motivation for learning, assessment, and feedback. METHOD: The authors performed a qualitative educational case study on the Transitioning in Internal Medicine Education Leveraging Entrustment Scores Synthesis (TIMELESS) program at the University of Cincinnati from October 2020 through March 2022. Semistructured interviews were conducted with TIMELESS residents (n = 9) approximately every 6 months to capture experiences over time. The authors used inductive thematic analysis to develop themes and compared their findings with existing theories of learner motivation. RESULTS: The authors developed 2 themes: TIMELESS had variable effects on residents' motivation for learning and TIMELESS increased resident engagement with and awareness of the program of assessment. Participants reported increased motivation to learn and seek assessment, though some felt a tension between performance (e.g., advancement through the residency program) and growth (e.g., improvement as a physician). Participants became more aware of the quality of assessments they received, in part due to TIMELESS increasing the perceived stakes of assessment, and reported being more deliberate when assessing other residents. CONCLUSIONS: Resident motivation for learning, assessment, and feedback was impacted in ways that the authors contextualize using current theories of learner motivation (i.e., goal orientation theory and attribution theory). Future research should investigate how interventions, such as coaching, guided learner reflection, or various CBTVT implementation strategies, can help keep learners oriented toward mastery learning rather than toward performance.

Development of entrustable professional activities for post-doctorate pediatric dentistry education.

PURPOSE: To identify the core components of pediatric dentistry defining entrustable professional activities (EPAs) representing the profession. METHODS: Potential core components of pediatric dentistry and corresponding domains were identified through review of literature and existing pediatric dentistry standards. A modified Delphi technique was utilized to rate these candidate EPAs to achieve consensus around prioritized EPAs. RESULTS: Eleven participants participated in all three rounds of the Delphi. After three rounds, 16 candidate EPAs reached consensus for pediatric dentistry. Each EPA fell into one of four domains: "assessment and planning," "provision of care," "behavior guidance," and "professional development." An original candidate EPA focused on non-pharmacological behavior guidance was deemed too broad by the Delphi. This EPA was subsequently developed into three separate components on nitrous oxide analgesia, moderate sedation, and general anesthesia. CONCLUSIONS: Prioritized EPAs will help define the essential activities of the profession and provide a framework for creating assessments to ensure that graduating pediatric residents are ready for unsupervised practice.

Application of a Child Sex Trafficking Screening Tool in Patients with Abuse: A Retrospective Chart Review in the Pediatric Emergency Department.

INTRODUCTION: Human trafficking (HT) is a global problem that may affect children's health. In the United States, victims and children are at risk in most communities. History of abuse is a risk factor for HT. This study explored associations between pediatric patients with positive universal abuse screens and indicators from the commercial sexual exploitation of children/child sex trafficking (CSEC/CST) screening tool. METHOD: A retrospective chart review was conducted on random patients, aged 11-17 years, with positive universal abuse screens at emergency/urgent care departments in a large Midwest pediatric medical center in 2018. Documentation identifying at least two CSEC/CST screening tool indicators was abstracted from these records. Data analysis included descriptive statistics, univariate analyses, and correlations. RESULTS: Two or more indicators from the CSEC/CST screening tool were identified in 43% (n = 121). Age and history of running away were significant predictors for a patient having two or more CSEC/CST positive indicators. DISCUSSION: Targeted screening and interventions are needed to identify and help these vulnerable youth.

Resident Sensitive Quality Measures for General Pediatrics: Alignment With Existing Care Recommendations.

BACKGROUND: Resident sensitive quality measures (RSQMs) are a new patient-focused approach to performance assessment. Their alignment with existing performance constructs is underexplored. OBJECTIVE: Develop and prioritize RSQMs for outpatient general pediatrics in the areas of well visits (0-2 months, 1, and 5 years) and compare to the existing framework of the Bright Futures recommendations. METHODS: Categorical pediatric residents, general academic pediatric fellows, and general pediatric faculty participated in nominal group technique (NGT) and Delphi processes to generate and prioritize RSQMs for 3 different well-child visits of interest. The author team then compared the content of the final RSQMs to the American Academy of Pediatrics Bright Futures recommendations. RESULTS: From the list of 143 potential measures from the NGT groups, 15 RSQMs were prioritized for each well-child visit. RSQMs prioritized vaccine administration, documentation of screening, and medication management. Overall, RSQMs mapped well to Bright Futures recommendations. CONCLUSIONS: The alignment of the RSQMs with Bright Futures frameworks supports their contribution as process measures for informing resident performance. RSQMs created from this study should be used for future investigations into resident performance assessment and to drive personal improvement efforts.

Identifying Core Components of EPA Implementation: A Path to Knowing if a Complex Intervention Is Being Implemented as Intended.

PURPOSE: Competency-based assessment, using entrustable professional activities (EPAs), is rapidly being implemented worldwide without sufficient agreement on the essential elements of EPA-based assessment. The rapidity of implementation has left little time to understand what works in what circumstances and why or why not. The result is the attempted execution of a complex service intervention without a shared mental model for features needed to remain true to implementing an EPA assessment framework as intended. The purpose of this study was to identify the essential core components necessary to maintain integrity in the implementation of this intended intervention. METHOD: A formal consensus-building technique, the Delphi process, was used to identify core components for implementing an EPA-based assessment framework. Twelve EPA experts from the United States, Canada, and the Netherlands participated in this process in February and March 2020. In each Delphi round, participants rated possible core components on a scale from 1 to 6, with 1 reflecting the worst fit and 6 the best fit for EPA-based assessment implementation. Predetermined automatic inclusion and exclusion criteria for candidate core components were set at ≥ 80% of participants assigning a value of 5 or 6 and ≥ 80% assigning a value of 1 or 2, respectively. RESULTS: After 3 rounds, participants prioritized 10 of 19 candidate core components for inclusion: performance prediction, shared local mental model, workplace assessment, high-stakes entrustment decisions, outcomes based, value of the collective, informed clinical competency committee members, construct alignment, qualitative data, and entrustment decision consequences. The study closed after 3 rounds on the basis of the rankings and comments. CONCLUSIONS: Using the core components identified in this study advances efforts to implement an EPA assessment framework intervention as intended, which mitigates the likelihood of making an incorrect judgment that the intervention demonstrates negative results.

Implementing the Behavioral and Mental Health Entrustable Professional Activity: Insights for a Path Forward.

OBJECTIVE: Mental health problems in children are growing exponentially. General pediatricians, while in a unique position to address these issues as they arise, report they lack adequate training in assessing and managing behavioral/mental health (B/MH) problems. Underscoring the importance of this area, the American Board of Pediatrics has defined B/MH as one of only 17 foundational entrustable professional activities (EPAs) for general pediatric practice. Our goal was to explore the facilitators and barriers associated with implementing and assessing the B/MH EPA among pediatric residency programs in order to identify best practices and potential solutions to common barriers. METHODS: In this qualitative study, 18 key faculty members from 4 residency programs with 3 years' experience implementing and assessing their residents on the B/MH EPA were purposively sampled. Semistructured interviews were conducted with each participant, and interviews were analyzed utilizing a thematic analysis. RESULTS: Five themes were defined in the thematic analysis 1) B/MH training: who's responsible? 2) local context can serve as a barrier or facilitator, 3) B/MH may require longitudinal, integrated, and multidisciplinary training, 4) B/MH specialists: indispensable, yet a hurdle?, and 5) resident and faculty confidence and skill impact B/MH training. CONCLUSIONS: The need for robust training to prepare pediatric residency graduates to meet the needs of patients with B/MH problems has never been greater. This study provides important insights about gaps in B/MH training. These should inform future directions focused on addressing this need.

Facilitators and Inhibitors to Assessing Entrustable Professional Activities in Pediatric Residency.

OBJECTIVE: Research on entrustable professional activities (EPAs) has focused on EPA development with little attention paid to implementation experiences. This constructivist grounded theory study sought to begin filling this gap by exploring the experiences of pediatric residency programs with implementing EPA-based assessment. METHODS: Interviews with 19 program leader and clinical competency committee participants from 13 sites were held between January and July 2019. Participants were asked about their experiences with implementing EPA-based assessment. Data collection and analysis were iterative. RESULTS: Participants described a range of facilitators and inhibitors that influenced their efforts to implement EPA-based assessment. These fell into 4 thematic areas: 1) alignment of EPA construct with local views of performance and assessment, 2) assessing EPAs illuminates holes in the residency curriculum, 3) clinical competency committee structure and process impacts EPA-based assessment, and 4) faculty engagement and development drives ability to assess EPAs. Areas described as facilitators by some participants were noted to be inhibitors for others. The sum of a program's facilitators and inhibitors led to more or less ability to assess EPAs on the whole. Finally, the first area functions differently from the others; it can shift the entire balance toward or away from the ability to assess EPAs overall. CONCLUSION: This study helps fill a void in implementation evidence for EPA-based assessment through better understanding of facilitators and inhibitors to such efforts.

Development of Resident-Sensitive Quality Measures for Inpatient General Internal Medicine.

BACKGROUND: Graduate medical education (GME) training has long-lasting effects on patient care quality. Despite this, few GME programs use clinical care measures as part of resident assessment. Furthermore, there is no gold standard to identify clinical care measures that are reflective of resident care. Resident-sensitive quality measures (RSQMs), defined as "measures that are meaningful in patient care and are most likely attributable to resident care," have been developed using consensus methodology and piloted in pediatric emergency medicine. However, this approach has not been tested in internal medicine (IM). OBJECTIVE: To develop RSQMs for a general internal medicine (GIM) inpatient residency rotation using previously described consensus methods. DESIGN: The authors used two consensus methods, nominal group technique (NGT) and a subsequent Delphi method, to generate RSQMs for a GIM inpatient rotation. RSQMs were generated for specific clinical conditions found on a GIM inpatient rotation, as well as for general care on a GIM ward. PARTICIPANTS: NGT participants included nine IM and medicine-pediatrics (MP) residents and six IM and MP faculty members. The Delphi group included seven IM and MP residents and seven IM and MP faculty members. MAIN MEASURES: The number and description of RSQMs generated during this process. KEY RESULTS: Consensus methods resulted in 89 RSQMs with the following breakdown by condition: GIM general care-21, diabetes mellitus-16, hyperkalemia-14, COPD-13, hypertension-11, pneumonia-10, and hypokalemia-4. All RSQMs were process measures, with 48% relating to documentation and 51% relating to orders. Fifty-eight percent of RSQMs were related to the primary admitting diagnosis, while 42% could also be related to chronic comorbidities that require management during an admission. CONCLUSIONS: Consensus methods resulted in 89 RSQMs for a GIM inpatient service. While all RSQMs were process measures, they may still hold value in learner assessment, formative feedback, and program evaluation.

Use of Resident-Sensitive Quality Measure Data in Entrustment Decision Making: A Qualitative Study of Clinical Competency Committee Members at One Pediatric Residency.

PURPOSE: Resident-sensitive quality measures (RSQMs) are quality measures that are likely performed by an individual resident and are important to care quality for a given illness of interest. This study sought to explore how individual clinical competency committee (CCC) members interpret, use, and prioritize RSQMs alongside traditional assessment data when making a summative entrustment decision. METHOD: In this constructivist grounded theory study, 19 members of the pediatric residency CCC at Cincinnati Children's Hospital Medical Center were purposively and theoretically sampled between February and July 2019. Participants were provided a deidentified resident assessment portfolio with traditional assessment data (milestone and/or entrustable professional activity ratings as well as narrative comments from 5 rotations) and RSQM performance data for 3 acute, common diagnoses in the pediatric emergency department (asthma, bronchiolitis, and closed head injury) from the emergency medicine rotation. Data collection consisted of 2 phases: (1) observation and think out loud while participants reviewed the portfolio and (2) semistructured interviews to probe participants' reviews. Analysis moved from close readings to coding and theme development, followed by the creation of a model illustrating theme interaction. Data collection and analysis were iterative. RESULTS: Five dimensions for how participants interpret, use, and prioritize RSQMs were identified: (1) ability to orient to RSQMs: confusing to self-explanatory, (2) propensity to use RSQMs: reluctant to enthusiastic, (3) RSQM interpretation: requires contextualization to self-evident, (4) RSQMs for assessment decisions: not sticky to sticky, and (5) expectations for residents: potentially unfair to fair to use RSQMs. The interactions among these dimensions generated 3 RSQM data user profiles: eager incorporation, willing incorporation, and disinclined incorporation. CONCLUSIONS: Participants used RSQMs to varying extents in their review of resident data and found such data helpful to varying degrees, supporting the inclusion of RSQMs as resident assessment data for CCC review.

Resident-Sensitive Quality Measures in the Pediatric Emergency Department: Exploring Relationships With Supervisor Entrustment and Patient Acuity and Complexity.

PURPOSE: This study explores the associations between resident-sensitive quality measures (RSQMs) and supervisor entrustment as well as between RSQMs and patient acuity and complexity for encounters in the pediatric emergency department (PED) in which residents are caring for patients. METHOD: Pediatric residents rotating through Cincinnati Children's Hospital Medical Center PED as well as supervising pediatric emergency medicine faculty and fellows were recruited during the 2017-2018 academic year for the purpose of collecting the following data from the residents' patient encounters for 3 illnesses (acute asthma exacerbation, bronchiolitis, and closed head injury [CHI]): supervisor entrustment decision rating, RSQMs relevant to the care provided, and supervisor patient acuity and complexity ratings. To measure the association of RSQM composite scores with the other variables of interest, mixed models were used. RESULTS: A total of 83 residents cared for 110 patients with asthma, 112 with bronchiolitis, and 77 with CHI. Entrustment decision ratings were positively associated with asthma RSQM composite scores (beta coefficient = 0.03; P < .001). There was no significant association between RSQM composite scores and entrustment decision ratings for bronchiolitis or CHI. RSQM composite scores were significantly higher when acuity was also higher and significantly lower when acuity was also lower for both asthma (P < .001) and bronchiolitis (P = .01). However, RSQM composite scores were almost identical between levels of acuity for CHI (P = .94). There were no significant differences in RSQM composite scores when complexity varied. CONCLUSION: This study found limited associations between RSQM composite scores and entrustment decision ratings but offers insight into how RSQMs could be used for the purposes of resident assessment and feedback.

Initial Implementation of Resident-Sensitive Quality Measures in the Pediatric Emergency Department: A Wide Range of Performance.

PURPOSE: A lack of quality measures aligned with residents' work led to the development of resident-sensitive quality measures (RSQMs). This study sought to describe how often residents complete RSQMs, both individually and collectively, when they are implemented in the clinical environment. METHOD: During academic year 2017-2018, categorical pediatric residents in the Cincinnati Children's Hospital Medical Center pediatric emergency department were assessed using RSQMs for acute asthma exacerbation (21 RSQMs), bronchiolitis (23 RSQMs), and closed head injury (19 RSQMs). Following eligible patient encounters, all individual RSQMs for the illnesses of interest were extracted from the health record. Frequencies of 3 performance classifications (opportunity and met, opportunity and not met, or no opportunity) were detailed for each RSQM. A composite score for each encounter was calculated by determining the proportion of individual RSQMs performed out of the total possible RSQMs that could have been performed. RESULTS: Eighty-three residents cared for 110 patients with asthma, 112 with bronchiolitis, and 77 with closed head injury during the study period. Residents had the opportunity to meet the RSQMs in most encounters, but exceptions existed. There was a wide range in the frequency of residents meeting RSQMs in encounters in which the opportunity existed. One closed head injury measure was met in all encounters in which the opportunity existed. Across illnesses, some RSQMs were met in almost all encounters, while others were met in far fewer encounters. RSQM composite scores demonstrated significant range and variation as well-asthma: mean = 0.81 (standard deviation [SD] = 0.11) and range = 0.47-1.00, bronchiolitis: mean = 0.62 (SD = 0.12) and range = 0.35-0.91, and closed head injury: mean = 0.63 (SD = 0.10) and range = 0.44-0.89. CONCLUSIONS: Individually and collectively, RSQMs can distinguish variations in the tasks residents perform across patient encounters.

The Power of Contribution and Attribution in Assessing Educational Outcomes for Individuals, Teams, and Programs.

Recent discussions have brought attention to the utility of contribution analysis for evaluating the effectiveness and outcomes of medical education programs, especially for complex initiatives such as competency-based medical education. Contribution analysis focuses on the extent to which different entities contribute to an outcome. Given that health care is provided by teams, contribution analysis is well suited to evaluating the outcomes of care delivery. Furthermore, contribution analysis plays an important role in analyzing program- and system-level outcomes that inform program evaluation and program-level improvements for the future. Equally important in health care, however, is the role of the individual. In the overall contribution of a team to an outcome, some aspects of this outcome can be attributed to individual team members. For example, a recently discharged patient with an unplanned return to the emergency department to seek care may not have understood the discharge instructions given by the nurse or may not have received any discharge guidance from the resident physician. In this example, if it is the nurse's responsibility to provide discharge instructions, that activity is attributed to him or her. This and other activities attributed to different individuals (e.g., nurse, resident) combine to contribute to the outcome for the patient. Determining how to tease out such attributions is important for several reasons. First, it is physicians, not teams, that graduate and are granted certification and credentials for medical practice. Second, incentive-based payment models focus on the quality of care provided by an individual. Third, an individual can use data about his or her performance on the team to help drive personal improvement. In this article, the authors explored how attribution and contribution analyses can be used in a complimentary fashion to discern which outcomes can and should be attributed to individuals, which to teams, and which to programs.

Key Factors in Clinical Competency Committee Members' Decisions Regarding Residents' Readiness to Serve as Supervisors: A National Study.

PURPOSE: Entrustment has become a popular assessment framework in recent years. Most research in this area has focused on how frontline assessors determine when a learner can be entrusted. However, less work has focused on how these entrustment decisions are made. The authors sought to understand the key factors that pediatric residency program clinical competency committee (CCC) members consider when recommending residents to a supervisory role. METHOD: CCC members at 14 pediatric residency programs recommended residents to one of five progressive supervisory roles (from not serving as a supervisory resident to serving as a supervisory resident in all settings). They then responded to a free-text prompt, describing the key factors that led them to that decision. The authors analyzed these responses, by role recommendation, using a thematic analysis. RESULTS: Of the 155 CCC members at the participating programs, 84 completed 769 supervisory role recommendations during the 2015-2016 academic year. Four themes emerged from the thematic analysis: (1) Determining supervisory ability follows from demonstrated trustworthiness; (2) demonstrated performance matters, but so does experience; (3) ability to lead a team is considered; and (4) contextual considerations external to the resident are at play. CONCLUSIONS: CCC members considered resident and environmental factors in their summative entrustment decision making. The interplay between these factors should be considered as CCC processes are optimized and studied further.

Developing Resident-Sensitive Quality Measures: Engaging Stakeholders to Inform Next Steps.

OBJECTIVE: Despite the need for quality measures relevant to the work residents complete, few attempts have been made to address this gap. Resident-sensitive quality measures (RSQMs) can help fill this void. This study engaged resident and supervisor stakeholders to develop and inform next steps in creating such measures. METHODS: Two separate nominal group techniques (NGTs), one with residents and one with faculty and fellow supervisors, were used to generate RSQMs for 3 specific illnesses (asthma, bronchiolitis, and closed head injury) as well as general care for the pediatric emergency department. Two separate Delphi processes were then used to prioritize identified RSQMs. The measures produced by each group were compared side by side, illuminating similarities and differences that were explored through focus groups with residents and supervisors. These focus groups also probed future settings in which to develop RSQMs. RESULTS: In the NGT and Delphi groups, residents and supervisors placed considerable focus on measures in 3 areas across the illnesses of interest: 1) appropriate medication dosing, 2) documentation, and 3) information provided at patient discharge. Focus groups highlighted hospital medicine and general pediatrics as priority areas for developing future RSQMs but also noted contextual variables that influence the application of similar measures in different settings. Residents and supervisors had both similar as well as unique insights into developing RSQMs. CONCLUSIONS: This study continues to pave the path forward in developing future RSQMs by exploring specific settings, measures, and stakeholders to consider when undertaking this work.